Perceptual Theory of Mind: An intermediary between visual salience and Noun / Verb Acquisition

We present computational models based on visual attention that learn object-name mappings and action semantics from simple 2D multi-agent visual streams co-occurring with word-separated utterance streams. We use no perceptual priors and the nominals are acquired by an early learner who has no syntactic knowledge. A late learner then uses the knowledge of nominals to identify actions referring to these arguments, and acquires the semantics of motion verbs like run or chase. Both early and late learners use visual attention to determine which parts of the scene are salient at the time of the utterance; we use a synthetic model of dynamic visual attention. Simple statistical measures based on joint probability are found sufficient to identify nominal participants from word separated input text, and a simple recurrent network is used to learn the verb semantics that encodes fine-grained image schemas as well as the argument structure as part of the semantic model.